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Background / Context: 

A central issue in nonexperimental studies is identifying comparable individuals to 
remove seleetion bias. One common way to address this seleetion bias is through propensity 
seore (PS) matching. PS methods use a model of the treatment assignment to reduce the 
dimensionality of the eovariate spaee and identify comparable individuals. PSs represent the 
conditional probability of treatment 

e(X) = P(Z = l|X) (1) 

where X are the observed eovariates and Z the treatment indieator. If adjustment on measured 
eovariates is sufficient for unbiased estimation of the treatment effeet, then so is adjustment on 
PSs (Rosenbaum & Rubin, 1983). As a result, PSs act as unidimensional balancing scores in 
which all the information relevant to balancing treatment assignment onXis extracted in e(X). 
Aecordingly, eases with similar PS values but different treatment assignments ean serve as 
estimates of the counterfaetual. 

In parallel to the PS, reeent literature has developed the prognosis score (PG) to construct 
models of the potential outeomes (Hansen, 2008). Whereas PSs summarize eovariates' 
assoeiation with the treatment groups, PGs summarize eovariates' associations with the potential 
outeomes. PGs eollapse a high dimensional eovariate space into a unidimensional measure 
summarizing covariates' assoeiations with the potential outeomes. PGs are eonstructed by 
modeling the outcome-covariate relations in the control group 

h^(X) = p(7<‘»|A) (2) 

where 1^'’^ are the potential outcomes for the control group and X are the measured eovariates. 
Like PSs, if adjustment on measured eovariates is suffieient for unbiased estimation of the 
treatment effect, then so is adjustment on PGs (Hansen, 2008). PGs thus provide a 
unidimensional balaneing seore in which all the information relevant to balaneing potential onX 
is extracted in W(X). PGs also induce a balancing property similar to that of PSs. Conditioning on 
PGs brings about prognostic balance among the eovariate distributions across potential outcomes 

( 3 ) 

The advantage of PGs is that they help to constrain variation in the outcome due to sources other 
than the treatment thereby favorably redueing both bias and variance of treatment effect 
estimators. Because PGs are estimated as a funetion of the eovariates using only the eontrol 
observations, PGs postpone eommitment to a functional form for both potential outeomes 
without risk of inducing bias associated with use of outcomes in the design stage (laeus et ah, 
2009). The separation of control and treatment groups is, in part, what differentiates PGs and 
helps them outperform other estimators (Hansen, 2006). 

Purpose / Objective / Research Question / Focus of Study: 

In observational studies where the treatment seleetion meehanism is not eompletely 
known, PGs ean be seen as an alternative or, more likely, a eomplement to PSs (Hansen, 2008). 
In this study, we explored the utility of combining adjustment on PGs and PSs by successively 
matching on these seores to examine the extent to which full matching on PSs within strata 
defined by PGs outperforms alternative PS and PS & PG matehing sehemes. 

Significance / Novelty of study: 

Prior literature has largely considered complementary adjustment for PSs and PGs in the 
form of Mahalanobis metrie matehing. Matehing treated and eontrol oases along oombinations 
defined by Mahalanobis distanoe attempts to equally weight the scores. With Mahalanobis 
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distances, net distanees between eases are formed by the equal adding up of the varianee 
normalized squared distanees of the seores. When estimated seores re fleet similar levels of 
preeision and importanee, Mahalanobis distanees are an optimal way of measuring similarities 
among treated and eontrol eases. 

In eontrast, when the estimated PGs and PSs differ in their preeision and meaningfulness, 
the squaring of distanees potentially allows one seore to obseure information provided by the 
other. Rather than define diserepaneies between potential matehes by eombining PGs and PSs 
into a single index using Mahalanobis distanee, we eonsidered first stratifying on PGs and 
subsequently matehing on PSs within prognostie strata. In this way, our approaeh tries to imitate 
a bloek randomized study whereby subjeets are stratified along the PGs and PSs. However, by 
seleeting finer matehing on the PSs, we privilege the bias reduetion properties of the PS. By 
seleeting eoarser subelassifieation on the PGs, we used them as a seeondary tool to further 
reduee error varianee. In this way, our approaeh suggests that adjustment on PSs is 
fundamentally more important and sound than adjustment along PGs. 

Statistical, Measurement, or Econometric Model: 

Unto themselves, PGs have several important limitations that are not shared with PSs. 

For instanee, although both summaries require the nontrivial assumption that all eonfounding 
variables have been observed, the asymmetrie estimation of PGs neeessitates an additional 
assumption; all effeet modifieations are eaptured in the prognostie model. In other words, use of 
PGs in isolation additionally assumes the suffieieney of the prognostie model for the eontrol 
potential outeomes is transferable to treated potential outeomes. Moreover, whereas PS balanee 
ean be appraised aeross the entire sample, PG balanee ean only be appraised in the eontrol 
sample. However, when paired with PSs, PGs have been shown to reduee both bias and varianee 
of estimated treatment effeets (Hansen, 2006). 

To integrate PGs and PSs, adjustment has largely eome through Mahalanobis distanees. 
Speeifieally, the two seores are eombined into a single index using 

(4) 

where X are the PGs and PSs with means fi and sample eovarianee matrix S. The eomparability 
of all eontrol and treatment eases is thus defined by a single index. Dm- Matehing along this 
eombined index has shown to be effeetive at both redueing bias and varianee (Hansen, 2006). 
Use of PGs in eombination with PSs through Mahalanobis metrie matehing tends to be 
partieularly favorable in the presenee of nonlinear relationships beeause matehing, to some 
extent, alleviates assumptions of funetional form. Aeeordingly, this type of matehing has been 
shown to outperform both regression models and estimators whieh base matehing only on PSs 
(Hansen, 2006). 

Combining PGs and PSs into a single index, however, may not always be well suited for 
investigating treatment effeets. In partieular, matehing on unified Mahalanobis distanees leaves 
open the potential for one seore to dominate the other. Although Mahalanobis distanees are 
designed to take into aeeount differenees in the distributions of the seores through the eovarianee 
matrix, the eontribution of seores to the distanees, and thus matehes, ean be obseured. 

Poor performanee of Mahalanobis metrie matehing has been noted in the presenee of rare 
events (e.g., Rubin & Thomas, 2000). Mahalanobis metrie matehing tends to plaee undue 
emphasis on variables with low prevalenee rates beeause their varianee tends to be so small. As a 
result, when eombining PGs and PSs through Mahalanobis distanee, it is possible that the 
distanee will favor elose matehes on the PGs and more eoarse matehes on PSs. The implieations 
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of this weakness may be partieularly aeute for studies investigating rare diehotomous outeomes. 
Conversely, when the eontrol to treatment eases ratio is large, matehes may be dominated by 
PSs. 

Similarly, when scores contain outliers or contain excessive noise, the estimated variance 
can diminish its contributions to Mahalanobis distance. Such sensitivity may be important when 
outcomes in the control group have a low signal to noise ratio. These difficulties may be further 
exacerbated because PSs and PGs are constructed using different samples; PSs with the full data 
and PGs with the control only. As a result, the variance in PGs may often reflect a larger portion 
of error than do PSs. In part, the poor performance of Mahalanobis metric matching, especially 
in high covariate space, is what in part led to the development of the PS (e.g., Rubin & Thomas, 
2000). 

There are also critical differences in the balancing properties of PGs and PSs. The 
balancing property of PSs can be explicitly appraised and ensured for the full sample. In contrast, 
prognostic balance can only be examined and ensured for the control sample. And although 
balance is a necessary but not sufficient condition for bias reduction, the authentication of PG 
balance using only control cases inherently produces balance and results that are more sensitive 
to misspecifications. Consequently, the balancing properties of PSs and their implications would 
seem to be more robust and important than those of PGs. 

Finally, our review of the literature suggested that bias reduction should be 
fundamentally more important than variance reduction in observational studies. Because PGs are 
primarily used to reduce error variance whereas PSs are primarily used to reduce bias, our review 
suggests that PSs and PGs should not be equally weighted. 

To reduce bias and secondarily to reduce variance while maintaining both propensity and 
prognostic balance, we propose PS matching within prognostic strata. In other words, to improve 
the robustness of simultaneous adjustment on PGs and PSs, we propose stratification on PGs 
followed by within stratum matching on PSs. To this end, matching on PSs within prognostic 
strata privileges PSs over PGs. In other words, our approach trades coarser matching on PGs for 
finer matching on PSs. 


Usefulness / Applicability of Method: 

As a preliminary examination of the utility of this approach, we performed multiple 
Monte Carlo simulations and highlight one for brevity. This simulation examined the 
performance of three estimators: 

1) Full matching with a 0.1 caliper on PSs only 

2) Full matching on PGs and PSs using Mahalanobis distance 

3) Full matching on PSs within PG strata (quintiles) 

We describe the simplest simulation in which we generated ten covariates (X) using 
independent standard normal distributions with sample sizes of 1000. We designed the treatment, 

Z, as the realization of a dichotomous variable given covariates which followed 

10 

logit(P(Z = D) = A + y (5) 

m=l 


with set at 0.3. The true outcome model was simulated as a nonlinear function with treatment 
by covariate interactions so that PGs constructed using only the control group would not be 
sufficient for the treatment group (different functional forms) 

10 


Y,=/3, + SZ, + 


+ 


( 6 ) 


m=l 
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where eoeffieients were analogous to the treatment model and the average treatment effeet, S, 
was 0.5. 

We estimated PSs as 

10 

logit{P{Z = l)) = fi, + Y,/^^X^, ( 7 ) 

m=l 

and PGs by estimating and extrapolating the fit using 

10 

r'"’ = A+[£/?A,,]+s; (8) 

m=l 

Subsequently, using full matehing with a 0.1 PS ealiper, units were matehed on (1) PSs; (2) PGs 
and PSs through Mahalanobis distanees; and (3) PGs and PSs by first stratifying along quintiles 
of PGs and then matehing units on PSs within strata. To estimate the treatment effeet for eaeh 
data set and approaeh, we eombined PS/PG adjustment with linear regression (e.g. Hirano & 
Imbens, 2002). Using mateh indieators, we modeled the outeome as: 

i' = A+<5Z,+X/?,M„+£, (9) 

9=1 

where S is the estimated treatment, M is the matehed group, and Q is the number of matehed 
groups minus one. To eompare the estimates, we used the results of the Monte Carlo simulations 
to estimate the bias and mean-squared error (MSB) of eaeh approaeh. We estimated these 
quantities using: 

^ias = — ( 10 ) 

1 ^ 

and MSE = —y^iS,-Sf (11) 

where N represents the number of simulated data sets (1000). 

Findings / Results: 

The results suggested ineorporating PGs into PS matehing via PG stratifieation deereased 
bias by about 11% when matehing on the PS only (1) and by about 7% when matehing on PG & 
PS using Mahalanobis distanee (2) (Table 1). Similarly, using the new approaeh redueed the 
varianee of the estimator by 6% as eompared with PS matehing only and 10% as eompared with 
PS & PG Mahalanobis matehing. Together, we found that the new method redueed mean- 
squared error by about 17% and 1 1% as eompared with PS matehing and PS & PG Mahalanobis 
matehing. 

Conclusions: 

The results suggested that adjustment using both PSs and PGs provides two avenues by 
whieh to bloek bias and reduee varianee. The gains demonstrated in this very limited simulation 
suggest that strategies whieh reduee estimator variation while maintaining bias reduetion may 
improve treatment effeet estimates in ways parallel to ehoiees in experimental design. Among 
many limitations of this very small initial investigation, the sequential use of PGs and PSs may 
have the undesirable effeet of redueing the availability of PS matehes within strata. In turn, the 
quality of PS matehes, and thus bias reduetion, may be negatively impaeted. In understanding the 
utility of ineorporating PG adjustment alongside PS adjustment in unequal ways, it will be 
important to explieate the risks and benefits of these situations. 
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Appendix B, Tables and Figures 

Table 1: Comparison of different matching estimators 



PS Only 

PG & PS Using 
Mahalanobis 

PG & PS Using 
Matching Within 
Prognostic Strata 

Bias X 10^ 

8.64 

8.24 

7.70 

Variance x 10 

1.52 

1.43 

1.29 

Mean-squared Error x 10 

2.26 

2.11 

1.88 
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